ZipLine: an optimized algorithm for the elastic bulk synchronous parallel model
نویسندگان
چکیده
The bulk synchronous parallel (BSP) is a celebrated synchronization model for general-purpose computing that has successfully been employed distributed training of deep learning models. A shortcoming the BSP it requires workers to wait straggler at every iteration. Therefore, employing increases waiting time faster cluster and results in an overall prolonged time. To ameliorate this BSP, we propose ElasticBSP, aims relax its strict requirement with elastic by allowing delayed minimize ElasticBSP offers more flexibility adaptability during phase, without sacrificing accuracy trained model. realized algorithm named ZipLine, which consists two phases. First, estimates each worker end points future iterations run time, then one-pass over estimated all fast compute optimal point synchronization. We provide theoretical about correctness performance ZipLine algorithm. Furthermore, algorithmic implementation optimizations namely ZipLineOpt ZipLineOptBS, reduce complexity linearithmic thorough experimental evaluation demonstrates our proposed model, materialized optimized variants, converges higher than predominant BSP. focus paper on optimizing scheduling parameter server architecture. It orthogonal other types optimizations, such as rate optimization.
منابع مشابه
The Heterogeneous Bulk Synchronous Parallel Model
Trends in parallel computing indicate that heterogeneous parallel computing will be one of the most widespread platforms for computation-intensive applications. A heterogeneous computing environment o ers considerably more computational power at a lower cost than a parallel computer. We propose the Heterogeneous Bulk Synchronous Parallel (HBSP) model, which is based on the BSP model of parallel...
متن کاملParallel Superposition for Bulk Synchronous Parallel ML
The BSMLlib is a library for Bulk Synchronous Parallel programming with the functional language Objective Caml. It is based on an extension of the λ-calculus by parallel operations on a parallel data structure named parallel vector, which is given by intention. Those operations are flat and allow BSP programming in direct mode but it is impossible to express directly divide-and-conquer algorith...
متن کاملParallel Juxtaposition for Bulk Synchronous Parallel ML
The BSMLlib library is a library for Bulk Synchronous Parallel (BSP) programming with the functional language Objective Caml. It is based on an extension of the λ-calculus by parallel operations on a parallel data structure named parallel vector. An attempt to add a parallel composition to this approach led to a non-confluent calculus and to a restricted form of parallel composition. This paper...
متن کاملBulk Synchronous Parallel Algorithms for the External
Blockwise access to data is a central theme in the design of efficient external memory (EM) algorithms. A second important issue, when more than one disk is present, is fully parallel disk I/O. In this paper we present a simple, deterministic simulation technique which transforms certain Bulk Synchronous Parallel (BSP) algorithms into efficient parallel EM algorithms. It optimizes blockwise dat...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: Machine Learning
سال: 2021
ISSN: ['0885-6125', '1573-0565']
DOI: https://doi.org/10.1007/s10994-021-06064-w